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Abstract. We consider qualitative and quantitative verification problems for infinite- 
state Markov chains. We call a Markov chain decisive w.r.t. a given set of target states F 
if it almost certainly eventually reaches either F or a state from which F can no longer be 
reached. While all finite Markov chains are trivially decisive (for every set F), this also 
holds for many classes of infinite Markov chains. 

Infinite Markov chains which contain a finite attractor are decisive w.r.t. every set F. In 
particular, all Markov chains induced by probabilistic lossy channel systems (PLCS) con- 
tain a finite attractor and are thus decisive. Furthermore, all globally coarse Markov chains 
are decisive. The class of globally coarse Markov chains includes, e.g., those induced by 
probabilistic vector addition systems (PVASS) with upward-closed sets F, and all Markov 
chains induced by probabilistic noisy Turing machines (PNTM) (a generalization of the 
noisy Turing machines (NTM) of Asarin and Collins) . 

We consider both safety and liveness problems for decisive Markov chains. Safety: 
What is the probability that a given set of states F is eventually reached. Liveness: What 
is the probability that a given set of states is reached infinitely often. There are three 
variants of these questions. (1) The qualitative problem, i.e., deciding if the probability is 
one (or zero); (2) the approximate quantitative problem, i.e., computing the probability 
up-to arbitrary precision; (3) the exact quantitative problem, i.e., computing probabilities 
exactly. 

1. We express the qualitative problem in abstract terms for decisive Markov chains, 
and show an almost complete picture of its decidability for PLCS, PVASS and PNTM. 

2. We also show that the path enumeration algorithm of Iyer and Narasimha terminates 
for decisive Markov chains and can thus be used to solve the approximate quantitative 
safety problem. A modified variant of this algorithm can be used to solve the approximate 
quantitative liveness problem. 

3. Finally, we show that the exact probability of (repeatedly) reaching F cannot be 
effectively expressed (in a uniform way) in Tarski-algebra for either PLCS, PVASS or 
(P)NTM (unlike for probabilistic pushdown automata). 
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1. Introduction 

Verification of infinite systems. The aim of model checking is to decide algorithmically 
whether a transition system satisfies a specification. Specifications which are formulated as 
reachability or repeated reachability of a given set of target states are of particular interest 
since they allow to analyze safety and progress properties respectively. In particular, model 
checking problems w.r.t. w-regular specifications are reducible to the repeated reachability 
problem. 

A main challenge has been to extend the applicability of model checking to systems with 
infinite state spaces. Algorithms have been developed for numerous models such as timed 
automata, Petri nets, pushdown systems, lossy channel systems, parameterized systems, 
etc. 

Probabilistic systems. In a parallel development, methods have been designed for the 
analysi s of models with stochastic behaviors (e.g. [LS821 lHS84l IVar851 ICY881 ICY951 IHK971 
CSS03]). The motivation is to capture the behaviors of systems with uncertainty such as 
programs with unreliable channels, randomized algorithms, and fault-tolerant systems. The 
underlying semantics for such models is often that of a Markov chain. In a Markov chain, 
each transition is assigned a probability by which the transition is performed from a state 
of the system. In probabilistic model checking, three classes of problems are relevant: 

• The qualitative problem: check whether a certain property <I> holds with probability one 
(or zero). 

• The approximate quantitative problem: compute the probability p of satisfying a given 
property $ up-to arbitrary precision, i.e., for any pre-defined error margin e > 0, compute 
a value p' s.t. p' < p < p' + e. 

• The exact quantitative problem: compute the probability p of satisfying a given property 
$ exactly and decide exact questions, e.g., if p > 0.5. 

Recently, several attempts have been made to consider systems which combine the above 
two features, i.e., systems which are infinite-state and which exhibit probabilistic behavior. 
For instance the works in |Rab031 iBSMl IAB031 lBE99l IPN971 lA"BI.T00| consider Probabilistic 
Lossy Channel Systems (PLCS): systems consisting of finite-state processes, which commu- 
nicate through channels which are unbounded and unreliable in the sense that they can 
spontaneously lose messages. The motivation for these works is that, since we are dealing 
with unreliable communication, it is relevant to take into consideration the probability by 
which messages are lost inside the channels. The papers [EKM04} IEKM051 IEKM061 IEY05b| 
EY05a, EE04, EY05cJ consider probabilistic pushdown automata (recursive state machines) 
which are natural models for probabilistic sequential programs with recursive procedures. 

Our contribution. Here we consider more abstract conditions on infinite Markov chains. 
We show how verification problems can be solved for Markov chains with these conditions 
and that several infinite-state probabilistic process models satisfy them. In particular, we 
consider probabilistic lossy channel systems (PLCS), probabilistic vector addition systems 
with states (PVASS) and probabilistic noisy Turing machines (PNTM). 

Let F be a given set of target states in a Markov chain, and F the set of states from 
which F cannot be reached, i.e., F := {s \ s -f^> F} = Pre*(F). We call a Markov chain 
decisive w.r.t. a given set of target states F if it almost certainly (i.e., with probability 1) 



DECISIVE MARKOV CHAINS 



3 



eventually reaches either F or F. In other words, decisiveness means that if F is always 
reachable then it will almost certainly be reached. 

While all finite Markov chains are trivially decisive (for every set F), this also holds for 
several classes of infinite-state Markov chains. 

It is not a meaningful question if the decisiveness property is decidable for general 
Markov chains. For finite Markov chains the answer is always yes, and for general infinite 
Markov chains the problem instance is not finitely given, unless one restricts to a partic- 
ular subclass. For some such subclasses decisiveness always holds, while for others (e.g., 
probabilistic pushdown automata (PPDA)) it is decidable (see below). 

• Markov chains which contain a finite attractor. An attractor is a set of states which is 
eventually reached with probability one from every state in the Markov chain. Examples 
of Markov chains with finite attractors are all Markov chains induced by probabilistic 
lossy channel systems (PLCS). 

We show that infinite Markov chains which contain a finite attractor are decisive w.r.t. 
every set F. 

• Markov chains which are globally coarse. A Markov chain is globally coarse w.r.t. F if 
there exists some a > such that, from every state, the probability of eventually reaching 
the set F is either zero or > a. Global coarseness w.r.t. F also implies decisiveness w.r.t. 
F. We consider two probabilistic process models which induce globally coarse Markov 
chains. 

- Any probabilistic vector addition system with states (PVASS) with an upward-closed 
set of final states F induces a globally coarse Markov chain. 

- Noisy Turing machines (NTM) have been defined by Asarin and Collins [AC05J. These 
are Turing machines where the memory tape cells are subject to 'noise', i.e., random 
changes. We consider probabilistic noisy Turing machines (PNTM), a generalization of 
noisy Turing machines (NTM) where the transition steps are also chosen probabilisti- 
cally. Probabilistic noisy Turing machines induce globally coarse Markov chains w.r.t. 
every set F defined by a set of control-states. 

• Another subclass of infinite Markov chains are those induced by probabilistic push- 
down automata (PPDA; also called recursive state machines) |EKM04t IEKM051 IEKM06| 
IEY05bl [EY05a[ EE04, EY05cj . These infinite Markov chains are not decisive in general. 
However, it follows directly from the results in [EKM06J that decisiveness is decidable for 
PPDA, provided that the set of final states F is effectively regular. 

The focus of this paper are the classes PLCS, PVASS and PNTM, not PPDA. We strive 
to be as general as possible and use only the weak condition of decisiveness. We do not 
advocate the use of our algorithms for PPDA, even for those instances which are decisive. 
Since PPDA is a special class with a particular structure, specialized algorithms like those 
described in [EKM041 IEKM051 IEKM061 IEY05bl IEY05al IEE04L IEY05cj are more suitable 
for it. However, we show in Section [9] that the techniques used for analyzing PPDA cannot 
be applied to PLCS, PVASS or PNTM. 

We consider both qualitative and quantitative analysis for decisive Markov chains. The 
main contributions of the paper are the following. 

• The qualitative reachability problem, i.e., the question if F is reached with probability 
1 (or 0). For decisive Markov chains, this problem is equivalent to a question about the 
underlying (non-probabilistic) transition system. 
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PLCS 


PVASS 


PNTM 


Approximate 
Prob M {s lm t (= OF) 


Solvable when i* 1 
is effectively 
representable; 
see Theorem IT 41 


Solvable when F 
is upward-closed; 

see Tbenrem IT Hi 


Solvable when F 
is defined by 
control-states; 
see Tbenrem IT 51 


Approximate 
Prob M (s„ ut (= OOF) 


Solvable when F 
is effectively 
representable; 
see Theorem 18.31 


Open problem 


Solvable when F 
is defined by 
control-states; 
see Theorem 18.41 


Compute the exact 
Prob M (simt h OF), or 
Prob M {sinit h nOF)0 


Not constructible 
when F is defined 
by control-states; 
see Theorem 19.3)/ 
Remark 19.51 


Not constructible 
when F is defined 
by control-states; 
see Theorem 19. f|/ 
Remark 19.51 


Not constructible 
when F is defined 
by control-states; 
see Theorem 19.4)/ 
Remark 19.51 



Table 1: Computability results for quantitative problems 



°A11 results here concern the effective expressibility of the probability in Tarski-algebra. 

For PVASS, the decidability of this question depends on the set of target states F. It is 
decidable if F is defined by a set of control-states, but undecidable if F is a more general 
upward-closed set of configurations. This is in contrast to all known decidability results 
for other models such as non-probabilistic VASS, and PLCS, where the two problems can 
effectively be reduced to each other. 

For both PLCS and PNTM, the qualitative reachability problem is generally decidable. 
In particular for PLCS, although this was already shown in [AR03, BS03], our construc- 
tion is more abstract and simpler. In particular, our algorithm does not require explicit 
construction of the attractor as in [AR03[ [BS03j. 

• The qualitative repeated reachability problem. 

If a Markov chain is decisive w.r.t. F then the question whether F will be visited 
infinitely often with probability 1 is equivalent to a simple question about the underlying 
transition graph, which is decidable for PVASS, PLCS and PNTM. For PVASS, the 
decidability of probabilistic repeated reachability is surprising, given the undecidability 
of probabilistic simple reachability above. 

If a Markov chain is decisive w.r.t. both F and F then the question whether F will 
be visited infinitely often with probability is equivalent to another question about the 
underlying transition graph. The precondition holds for all Markov chains with a finite 
attractor (such a PLCS) since they are decisive w.r.t. every set, and the question is 
decidable for PLCS. For PNTM, we show that if F is defined by a set of control-states 
then so is F. Since PNTM induce globally coarse Markov chains w.r.t. any set defined 
by control-states, the question is also decidable. 

However, for PVASS, decisiveness w.r.t. F does not generally imply decisiveness w.r.t. 
F and thus our algorithm is not always applicable. For PVASS, decidability of the 
question whether F is visited infinitely often with probability is an open problem. 

• To approximate the probability of eventually reaching F, we recall an algorithm from 
[IN97] which was also used in |Rab03j for PLCS. We show that the algorithm can be used 
to solve the problem for all decisive Markov chains (in particular also for both PVASS 
and PNTM). 
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PLCS 


PVASS 


PNTM 


Approximate 
Prob M {s lm t (= OF) 


Solvable when F 
is effectively 
representable; 
see Theorem IT 41 


Solvable when F 
is upward-closed; 

see Tbenrem IT Hi 


Solvable when F 
is defined by 
control-states; 
see Tbenrem IT 51 


Approximate 
Prob M (s„ ut (= OOF) 


Solvable when F 
is effectively 
representable; 
see Theorem 18.31 


Open problem 


Solvable when F 
is defined by 
control-states; 
see Theorem 18.41 


Compute the exact 
Prob M (simt h OF), or 


Not constructible 
when F is defined 
by control-states; 
see Theorem 19.3)/ 
Remark 19.51 


Not constructible 
when F is defined 
by control-states; 
see Theorem 19. f|/ 
Remark 19.51 


Not constructible 
when F is defined 
by control-states; 
see Theorem 19.4)/ 
Remark 19.51 



Table 2: Computability results for quantitative problems 



°A11 results here concern the effective expressibility of the probability in Tarski-algebra. 

Furthermore, we show that a minor modification of the algorithm yields an algorithm 
for approximating the probability of visiting F infinitely often for all Markov chains which 
are decisive w.r.t. F and F. In particular this works for all Markov chains with a finite 
attractor, such as PLCS. This is a more abstract, general and simpler solution than the 
result for PLCS in [Rab03j. However, it does not yield precise complexity bounds as 
|Rab03j . 

• The question if the exact probability of (either eventually, or infinitely often) reaching 
F in PLCS is expressible by standard mathematical functions was stated as an open 
problem in [Rab03] . We provide a partial answer by showing that for PVASS, PLCS and 
(P)NTM, this probability cannot be effectively expressed (in a uniform way) in Tarski- 
algebra, the first-order theory of the reals (H, + ,*,<). (By 'in a uniform way' we mean 
that quantitative parameters in the system should be reflected directly by constants in the 
Tarski-algebra- formula.) This is in contrast to the situation for probabilistic pushdown 
automata for which these probabilities can be effectively expressed, in a uniform way, in 
(R, +,*,<) [EKM041 IEKM061 IEY05bl lEEOl] . 

2. Transition Systems and Markov Chains 

We introduce some basic concepts for transition systems and Markov chains. Let N and 
Q>o denote the set of natural numbers (including 0) and non- negative rational numbers, 
respectively. 

2.1. Transition Systems. A transition system T is a tuple (S, — ►) where S is a (po- 
tentially) infinite set of states, and — ► is a binary relation on S. We write s — ► s' for 
(s, s') € — ► and let Post(s) := {s'| s — * s'}. A run p (from sq) of T is an infinite sequence 
sqS\ ... of states such that Sj — ► Sj+i for % > 0. We use p(i) to denote and say that 
p is an s-run if p(0) = s. A path is a finite prefix of a run. We assume familiarity with 
the syntax and semantics of the temporal logic CTL* [CGP99J. We use (s |= 0) to denote 
the set of s-runs that satisfy the CTL* path-formula (p. For s G S and Q C S, we say 
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that Q is reachable from s if s \= 30Q. For Qi,Q2 C S, we use Q\ Before Q2 to denote 
the CTL formula 3 (-1Q2 W Qi), i-e., there exists a run which reaches a state in Q\ with- 
out having previously passed through any state in Q2. Given a set of states F C S, we 
define Pre*{F) := {s' | 3s G F : s' — — > s} as the set of its predecessors. Furthermore, let 
F := Pre*(F) = {s| s 30F}, the set of states from which F is not reachable. For s £ S 
and F C £, we define the distance distp(s) of s to -F to be the minimal natural number n 
with s — —> F. In other words, distp(s) is the length of the shortest path leading from s to 
F. In case s € F, we define distp(s) = 00. A transition system T is said to be of span N 
with respect to a given set F if for each s £ S we either have distp(s) < N or distp(s) = 00. 
We say that T is finitely spanning with respect to a given set -F if T is of span iV w.r.t. F 
for some iV > 0. A transition system T = (S, — ►) is said to be effective w.r.t. a given set 
F if for each s € S, we can (1) compute elements of the set Post(s) (notice that this implies 
that T is finitely branching); and (2) check whether s \= 30F. 

2.2. Markov Chains. A Markov chain Ai is a tuple (S, P) where S is a (potentially 
infinite) set of states, and P : S X S — *■ [0, 1], such that Yls'^s ^ > ( s ' s ') = ^> ^ or eac h s £ S. 
A Markov chain induces a transition system, where the transition relation consists of pairs 
of states related by positive probabilities. In this manner, concepts defined for transition 
systems can be lifted to Markov chains. For instance, for a Markov chain M, a run of Ai 
is a run in the underlying transition system, and Ai is finitely spanning w.r.t. given set F 
if the underlying transition system is finitely spanning w.r.t. F, etc. 

Consider a state sq of a Markov chain A4 = (S,P). On the sets of so- runs ) the proba- 
bility space (f2, A, Probj^) is defined as follows (see also [KSK66J): Q = sqS w is the set of 
all infinite sequences of states starting from so, A is the cr-algebra generated by the basic 
cylindric sets D u = uS^, for every u £ sqS*, and the probability measure Probjn is defined 
by Prob_M(D u ) = Y\ i=0 n _ ± P(s{, Sj+i) where u = soSi...s n ; this measure is extended in a 
unique way to the elements of the cr-algebra generated by the basic cylindric sets. 

Given a CTL* path-formula c/>, we use (s |= c/>) to denote the set of s-runs that satisfy 
4>. We use Prob_M (s |= c/>) to denote the measure of the set of s-runs (s |= <f>) (which is 
measurable by |Var85j ). For instance, given a set F C S, Probj^ (s |= OF) is the measure of 
s-runs which eventually reach F. In other words, it is the probability by which s satisfies OF. 
We say that almost all runs of a Markov chain satisfy a given property (j> if Probjn (s \= </>) = 
1. In this case one says that (s |= <f>) holds almost certainly. 

3. Classes of Markov Chains 

In this section we define several abstract properties of infinite-state Markov chains: deci- 
siveness, the existence of a finite attractor, and global coarseness. We show that both the 
existence of a finite attractor and global coarseness imply decisiveness. In particular, all 
three properties hold trivially for finite Markov chains. 

In the rest of this section, we assume a Markov chain Ai = (S, P). 
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3.1. Decisive Markov Chains. 

Definition 3.1. Given a Markov chain A4 = (S, P) and a set of states F C S, we say that 
M is decisive w.r.t. F iff Pro6^(s |= OF V OF) = 1, for each s G S. 

In other words, the set of runs, along which F is always reachable but which never 
reach F, is almost empty (i.e., has probability measure zero). 

Similarly, we say that M. is strongly decisive w.r.t. F if Probjui 8 |= OF V OOF) = 1. 
Intuitively, this means that the set of runs along which F is always reachable and which 
visit F only finitely many times is almost empty. 

Lemma 3.2. Given a Markov chain Ai = (S,P) and a set F C S, A4 is decisive w.r.t. F 
iff it is strongly decisive w.r.t. F. 

Proof. Given a Markov chain A4 = (S, P) and a set F C S, we want to prove that Vs G 
6", Prob M {s \= <>F V OF) = 1 Vs G S, Prob M (s |= OF V DOF) = 1. This is equivalent 
to proving that 

Vs G S, Prob M (s |= D-.F A CHF) = Vs G 5, Prob M (s \= OD-.F A CHF) = 0. 

Let U be a set of sequences of states. U is called proper if no sequence in U is a prefix 
of another sequence in U. If all sequences in U are finite and start at the same state, we 
define P(U) := ProbjuiDu) where Djj = {uS^l u G U}. Given a proper set U of finite 
sequences (namely paths) ending all in the same state s c and a proper set V of possibly 
infinite sequences (runs) starting all from s c , we define U • V to be the set of all sequences 
us c v where us c G U and s c v G V. 

We now prove both implications of the required equivalence above. 

(<=) Observe that (s |= OD-iF) is the set of s-runs visiting F only finitely many 
times. In particular, the set of s-runs which never visit F is included in that set. This gives 
(s |= D-F) C (s |= On-iF). By intersection with (s |= n-F), the set of s-runs which never 
visit F, we obtain (s |= D->F A D->F) C (s |= Oa^F A D^F). 

By definition of the probability measure, we obtain Probes \= O-iF A O-iF) < 
Prob_M{ s \= OD-iF A D-iF) = for any s G 5, where the last equality follows from the 
assumption. 

(=>) Given a state s G S, we define the following sets of paths: 

II| r := {it 1 7r = s(F*F)V} where i > and r G F . 
Now, consider the following sets of runs: 

Vt > T 4 S := \J reT r? r where Vr e F Tf r := IIf r • (r |= A U—iF) 
Vi>0A s t := \J reT A| r where Vr e F A| r := n? r • (r5 w ) . 
Intuitively, A| is the set of s-runs which revisit F at least i times while Tf is the set of all 
s-runs which revisit F exactly i times and then never visit neither F nor F. Observe that 
for i = 0, Tq = (s |= D-F A CHF). It is straightforward to check that: 

(1) (Vi e N)(Vn,r 2 e F An ^ r 2 )(r? ri nrf r2 - 0) 

(2) (Vi,j GNA»^)(r? nr| = 0) 

(3) (Vi G N)(Vr G F)(Pro& A1 (rf r ) = F(n| 7 .)Pro6 A1 (r |= D-.F A D-F)) 
Therefore, it follows that for all i e N 

iVt»6^(T?) - ^ Pro6 ^( r ir-) - E ^(n? r )^o6 M (r h a-F A D-.F) = 

reT reT 
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where the first equality holds by (pQ). The second equality follows from ([3]), and the last 
from the fact that Ai is decisive w.r.t. F; i.e., for all rSS, Probj^ir \= O^F A 0—>F) = 0. 

Observe that f\=o A f £ ( s (= D< >^)- Therefore, (s (= OCHF) C U J = where 
for all ? > 0, — iA| is the set of s-runs revisiting F at most i — 1 times. For all i > 0, 
we have (^Af n (s |= O-^F)) C U}=o^j- By using this inclusion, property ([21), and the 
fact that T| has measure zero, we obtain Prob M (s \= OD->F A D->F) < Prob M (U^o^f) = 
ESo^M(Tf) = 0- ' □ 

3.2. Markov Chains with a Finite Attractor. 

Definition 3.3. Given a Markov chain Ai = (S, P), a set j4 C S 1 is said to be an attractor, 
if for each s € 5, we have ProbM ( s H ^^4) = 1, i-e., the set ^4 is reached from s with 
probability one. 

Lemma 3.4. A Markov chain Ai which has a finite attractor is decisive w.r.t. every set 
F C S. 

Proof. Fix a Markov chain Ai = (S, P) that has a finite attractor A, a state s and a set 
F C S. Recall that (s (= D-iF A D-ii^) denotes the set of s-runs which neither visit F 
nor F. All states s' visited by runs in (s (= D-iF A O—iF) satisfy s' (= BOF, since s' ^ -F. 
In particular this holds for the finitely many different s" G A visited by those runs. Let 
A' Q A denote the set of states from the attractor, visited by runs in (s (= D-iFAD-iF). For 
every s" € A' we define q s » := Probj^i(s" \= OF), and obtain a s ii > 0. By definition of an 
attractor, we obtain that A' is not empty. By finiteness of A (and thus A'), it follows that 
a := min s // G ^/ a s " > 0. Almost every run must visit A infinitely often, and only states in A' 
are visited by runs in (s |= n-<F A D-iF). Thus, Prob M {s \= n-<F A D-iF) < (1 - a)°° = 0. 
Finally, we obtain for each s G 5, Prob M (s \= OFVOF) = l-Prob M (s \= 0-<FAO-iF) = 1. 

□ 

3.3. Globally Coarse Markov Chains. 

Definition 3.5. A state s is said to be of coarseness (5 if for each s' G 5, P(s,s') > 
implies P(s, s') > (3. A Markov chain = (5, P) is said to be of coarseness (3 if each s £ S 
is of coarseness /?. We say that Ai is coarse if .M is of coarseness /?, for some (3 > 0. Notice 
that if Ai is coarse then the underlying transition system is finitely branching; however, the 
converse is not necessarily true. Given a Markov chain Ai = (S, P) and a set F C 5. We 
say that a Markov chain Ai = (S, P) is globally coarse w.r.t. F if there exists some a > 
s.t. Vs GS.(s^F)^ Prob M (s \= OF) > a. 

Lemma 3.6. If a Markov chain is coarse and finitely spanning w.r.t. a set F then it is 
globally coarse w.r.t. F. 

Proof. If a Markov chain is coarse (of coarseness /3 > 0) and finitely spanning w.r.t. a given 
set F (of span N) then it is globally coarse w.r.t. the same set F (define a := f3 N ). □ 
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Lemma 3.7. Given a Markov chain M. and a set F such that M. is globally coarse w.r.t. 
F, then A4 is decisive w.r.t. F. 

Proof. Assume a Markov chain M = (S,P), a state s and a set F C S such that A4 
is globally coarse w.r.t. F. All states s' visited by runs in (s \= O^F A O^F) satisfy 
s' \= 30F, because s' ^ F. Since M is globally coarse w.r.t. F there exists some universal 
constant a > s.t. Probj^ (V |= OF) > a for any s' which is visited by those runs. 
Therefore, Probj^is \= D->F A D-iF) < (1 — = 0. Finally, we obtain for each s G S, 
Prob M {s \= OF V OF) = 1 - Prob M {s \= D—iF A D-.F) = 1. □ 

4. System Models and their Properties 

We define three classes of infinite-state probabilistic system models and describe the induced 
Markov chains. 

4.1. Vector Addition Systems. A Vector Addition System with States (VASS) consists 
of a finite-state process operating on a finite set of unbounded variables each of which ranges 
over N. Formally, a VASS V is a tuple (S,X,T), where S is a finite set of control- states, X 
is a finite set of variables, and T is a set of transitions each of the form (si, op, S2), where 
si, S2 G S, and op is a mapping from X to the set {—1,0, 1}. A (global) state s is of the form 
(s,v) where s 6 S and v is a mapping from X to N. 

We use s and S to range over control-states and sets of control-states, respectively. 
On the other hand, we use s and S to range over states and sets of states of the induced 
transition system (states of the transition system are global states of the VASS). 

For vi,V2 G (X 1— > N), we use vi H V2 to denote that vi(x) < V2(x) for each x G X. We 
extend ^ to a relation on S x (X 1— ► N), where (sj,vi) if? (s2,V2) iff s\ = S2 and vi H V2. 

A set of global states F is upward-closed w.r.t. ^ iff 

(si,vi) G F A (si,vi) < (s 2 ,v 2 ) =>> (s 2 ,v 2 )GF 
Analogously, a set of global states F is downward-closed w.r.t. ^ iff 
(s2,v 2 ) G F A (si,vi) < (s 2 ,v 2 ) ^> (si,vi) G F 

The complement of an upward-closed set is downward-closed and vice-versa. 

For Q C S, we define a Q-state to be a state of the form (s, v) where s G Q. Notice that, 
for any Q C S, the set of Q-states is upward-closed and downward-closed with respect to 

It follows from Dickson's Lemma [Dicl3] that every infinite set of VASS configurations 
has only finitely many minimal elements w.r.t. ^. When we speak of an upward-closed 
set of VASS configurations, we assume that it is represented by its finitely many minimal 
elements. 

A transition t = (si,op, s 2 ) is said to be enabled at (si,vi) if vi(x) + op(x) > for 
each x G X. We define enabled(s,v) = {t| t is enabled at (s,v)}. In case t = (si,op, s 2 ) is 
enabled at (si,vi), we define t(si,vi) to be (s 2 ,v 2 ) where v 2 (x) = vi(x) + op(x) for each 
x G X. The VASS V induces a transition system (5, — ►), where S is the set of states, i.e., 
S = (S x (X 1— > N)), and (si, Vi) — > (s 2 , v 2 ) iff there is a t G T with (s 2 , v 2 ) = t(si, Vi). In 
the sequel, we assume, without loss of generality, that for all (s,v), the set enabled(s,v) is 
not empty, i.e., there is no deadlock. This can be guaranteed by requiring that from each 
control-state there is a self-loop not changing the values of the variables. 
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VASS are expressively equivalent to Petri nets [Pet81] . The only difference is that VASS 
explicitly mention the finite control as something separate, while Petri nets encode it as 
another variable in the vector. The reachability problem for Petri nets/VASS is decidable 
|May84| and a useful extension of this result has been shown by Jancar [Jan90j. In our 
VASS terminology this result can be stated as follows. 

Theorem 4.1. (^JanQOjJ Let (S,X, T) be a VASS with control- states S = {s\,...,Sj} and 
variables X = {x%, . . . ,x n }. A simple constraint logic is used to describe properties of global 
states (s,xi, . . . , x n ). Any formula $ in this logic is a boolean combination of predicates of 
the following form: s = S{ (the control-state is Si), X{ > k, or Xi < k, where k G N. 

In particular, all upward-closed sets of VASS states can be described in this logic. It 
suffices to specify that the global state must be larger or equal (in every variable) than some 
of the (finitely many) minimal elements of the set. Since this constraint logic is closed under 
negation, all downward-closed sets can also be described in it. 

Given an initial global state (s,v), and a constraint logic formula <!>, it is decidable if 
there exists a reachable state that satisfies $. 

Probabilistic VASS. A probabilistic VASS (PVASS) V is of the form (S,X,T,w), where 
(S, X, T) is a VASS and w is a mapping from T to the set of positive natural numbers. 
Intuitively we derive a Markov chain from V by assigning probabilities to the transitions of 
the underlying transition system. The probability of performing a transition t from a state 
(s, v) is determined by the weight w(t) of t compared to the weights of the other transitions 
which are enabled at (s, v). We define w(s, v) = ^2teenabled(s v) w ( t )- The PVASS V induces 
a Markov chain (S, P), where S is defined as for a VASS, and 

p// \ / „ u EjgOOj t(si,Vi) = (S 2 ,V 2 )} 

P((Sl,Vi),(S2,V 2 )) = -7- . 

W(Sl, Vij 

Notice that this is well-defined since w(si,vi) > by the assumption that there are no 
deadlock states. 

Remark 4.2. Coarseness of Markov ch ains induced by PVASS follows immediately from 
the definitions. It follows from results in jACJYKOO] (Section 4 and 7.2) that each Markov 
chain induced by a PVASS is effective and finitely spanning w.r.t. any upward-clos ed set of 



final markings F. VASS induce well-structured systems in the sense of [ACJYKOO] and the 



computation of the set of predecessors of an ideal (here this means an upward-closed set) 
converges after some finite number k of steps. This yields the finite span k w.r.t. F of our 
Markov chain derived from a PVASS. 

By applying Remark 14.21 and Lemma 13.61 we obtain the following theorem. 

Theorem 4.3. Each Markov chain A4, induced by a PVASS satisfies the following: 

• Ai is coarse. 

• A4 is effective, finitely spanning and globally coarse w.r.t. any upward-closed set of final 
states. 

This combined with Lemma 13.21 and Lemma 13.71 yields the following corollary. 

Corollary 4.4. Each Markov chain induced by a PVASS is decisive (and thus, by Lemma \3.SX 
strongly decisive) w.r.t. any upward-closed set of final states. 



DECISIVE MARKOV CHAINS 



11 



4.2. Probabilistic Lossy Channel Systems. A Lossy Channel System (LCS) consists 
of a finite-state process operationg on a finite set of channels, each of which behaves as a 
FIFO buffer which is unbounded and unreliable in the sense that it can spontaneously lose 
messages [ATMl ICFI96] . 

Probabilistic lossy channel systems (PLCS) are a generalization of LCS to a probabilistic 
model for message loss and choice of transitions. There exist several variants of PLCS which 
differ in how many messages can be lost, with which probabilities, and in which situations, 
and whether normal transitions are subject to non-deterministic or probabilistic choice. We 
consider a partial order on channel contents, defined by w\ < u>2 iff w\ is a (not necessarily 
continuous) substring of W2- 

The most common PLCS model is the one from [AR031 IBS031 IRab 03] . where each 
message in transit independently has the probability A > of being lost in every step, and 
the transitions are subject to probabilistic choice in a similar way as for PVASS. However, 
the definition of PLCS in |AR03[ IBS03j IRab03j assumes that messages can be lost only 
after discrete steps, but not before them. Thus, since no messages can be lost before the 
first discrete step, the set {s G S : s \= 30F} of predecessors of a given set F of target 
states is generally not upward-closed w.r.t. <. 

Here we assume a more realistic PLCS model where messages can be lost before and 
after discrete steps. This PLCS model is also closer to the classic non-probabilistic LCS 
model where also messages can be lost before and after discrete steps [AJ96} ICFI96] . So we 
obtain that the set {s G S : s (= 3C>F} is always upward-closed w.r.t. <. 

Definition 4.5. Formally, a PLCS is a tuple £ = (S, C, M, T, A, w) where S is a finite set of 
control-states, C is a finite set of unbounded fifo- channels, M is a finite set called the message 
alphabet, T is a set of transitions, < A < 1 is the message loss rate, and w : T — > N>o is the 
transition weight function. Each transition t G T is of the form s s', where s,s' G S 
and op is an operation of one of the following froms: c!m (send message m G M in channel 
c G C), c?m (receive message m from channel c), or nop (do not modify the channels). 

A PLCS C = (S, C, M, T, A, w) induces a transition system T = (S, — >), where S = 
S x (M*) c . That is, each state in S consists of a control-state and a function that assigns 
a finite word over the message alphabet to each channel called channel state. We define 
two transition relations — >a (called 'discrete transition') and — >i (called 'loss transition'), 
where — >d models the sending and receiving of messages and transitions taken in the 
underlying control structure, and — >\ models probabilistic losses of messages. 

The relation — >^ is defined as follows. If s = (s,x),s' = (s',x') G S, then there is a 
transition s — >d s' in the transition system iff one of the following holds: 

• s — > s and x = x ; 

• s ■—)■ s', x'(c) = x(c)m, and for all c' G C — {c}, x'(c') = x(c'); 

• s s', x(c) = mx'(c), and for all c' G C — {c}, x'(c') = x(c'). 

We assume, without loss of generality, that there are no deadlocks. This can be guaranteed 
by adding self-loops s s if necessary. 

If several discrete transitions are enabled at the same configuration then the next tran- 
sition is chosen probabilistically. The probability (P<j) that a particular transition is taken 
is given by the weight of this transition, divided by the sum of the weights of all currently 
enabled transitions. Since there are no deadlocks, this is well defined. 
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The transition — >i models probabilistic losses of messages. We extend the subword 
ordering < on words first to channel states x,x' : C -> M* by x < x' iff x(c) < x'(c) for all 
channels c£C, and then to the transition system states s = (s, x), s' = (s', x') € S by s < s' 
iff s = s', and x < x'. For any s = (s,x) and any x' such that x' < x, there is a transition 
s — >i (s,x'). The probability of loss transitions is given by Pi((s, x), (s, x')) = a-\ b -(l — A) c , 
where a is the number of ways to obtain x' by losing messages in x, b is the total number 
of messages lost in all channels, and c is the total number of messages in all channels of x'. 

The PLCS induces a Markov chain by alternating the probabilistic transition relations 
— >l and — >d in such a way that message losses can occur before and after every discrete 
transition, i.e., we consider transition sequences in — >[ ( — >^ — >/)*. 

We say that a set of target states F is effectively representable if a finite set F' can 
be computed s.t. F' f= F f, i.e., their upward-closures are e quivalent. (F or instance, any 



context-free language is effectively representable [Cou91].) In ACJYKOO it is shown that 
a Markov chain, induced by a PLCS is effective w.r.t. any effectively representable set F. 

However, many of our results do not strongly depend on a particular PLCS model. 
The only crucial aspects are the existence of a finite attractor in the induced Markov chain 
(most PLCS models have it) and the standard decidability results of the underlying non- 
probabilistic LCS [AJ961 ICFI 96] . In [AR03], it is shown that each Markov chain induced 
by a PLCS contains a finite attractor. 

Theorem 4.6. Each Markov chain induced by a PLCS contains a finite attractor and is 
effective w.r.t. any effectively representable set of global states F. 

From this result, Lemma 13.21 and Lemma 13.41 we obtain the following corollary. 

Corollary 4.7. Each Markov chain induced by a PLCS is decisive w.r.t. every set F and 
thus strongly decisive w.r.t. every set F. 

The PLCS models used here (and in [AB031 iBSMl IRab03| ) differ from the more un- 
realistic models considered previously in [ABIJOO, BE99]. In [BE99] at most one message 
could be lost during any step and in [ABIJOOj messages could be lost only during send 
operations. If one assumes a sufficiently high probability (> 0.5) of message loss for these 
models then they also contain a finite attractor. Another different PLCS model was stud- 
ied in [BS04]. It has the same kind of probabilistic message loss as our PLCS, but differs 
in having nondeterministic choice (subject to external schedulers) instead of probabilistic 
choice for the transitions, and thus does not yield a Markov chain, but a Markov decision 
process. Another difference is that the model of [BS04] allows (and in some cases requires) 
idle transitions which are not present in our PLCS model. However, for any scheduler, 
the PLCS model of [BS04] also has a finite attractor (w.r.t. the system-state, though not 
necessarily w.r.t. the state of the scheduler). 



4.3. Noisy Turing Machines. Noisy Turing Machines (NTM) were introduced in [AC05] . 
They are Turing Machines augmented by an additional parameter e > giving the noise 
level. Each transition of an NTM consists of two steps. First, in the noisy step the tape cells 
are subjected to noise. In this manner, each symbol in each tape may change independently 
and uniformly with probability e to any other symbol in the tape alphabet (possibly the 
same as before). Then, in the normal step, the NTM proceeds like a normal Turing machine. 

Probabilistic Turing Machines (PTM) [dLMSS56], which are Turing machines where 
transitions are random choices among finitely many alternatives, are more general than the 
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model of [AC05J. In fact, any NTM can be simulated by a PTM by adding extra steps 
where the machine makes a pass over the tapes changing the symbols randomly. However, 
as described below, general PTM do not satisfy our conditions. 

Probabilistic NTM. In this paper, we adopt the model of Probabilistic Noisy Turing 
Machines (PNTM) which are a generalization of NTM. In a PNTM, the transitions are 
similar to those of an NTM except that normal steps are subject to probabilistic choices. 
Formally, a PNTM Af is a tuple (S, E, r, M, T, e, w) where S is a finite set of control-states, E 
is the input alphabet, r 2 E U j) (where (j is the blank symbol) is the tape alphabet, M is the 
number of tapes, TCSxr M xSxr M x {—1,0, 1} M is the transition relation, e is the noise 
level and w : T — * N>o is the weight function. The probability of a transition t G T is given 
by comparing the weight w(t) to the weights of all possible alternatives. 

Assume a PNTM J\f = (S, E, r, M, T, e, w). A global state of N can be represented by a 
triple: (i) the control-state, (ii) the current time, and (hi) an M-tuple of tape configurations. 
A tape configuration is a triple: (i) the head position; (ii) a finite word u € T* representing 
the content of all cells visited by the head so far; and (in) a |u|-tuple of natural numbers, 
each representing the last point in time the head visited the corresponding cell. For a set 
Q C S, we let Q-states denotes the set of all global states whose control-states are in Q. 

For a PNTM M = (S, E, r, M, T, e, w), we use G(N) to denote the graph obtained from N 
by abstracting away the memory tapes. Formally, Q{M) is the tuple (S, T') where S is the set 
of control-states of the underlying PNTM M, and T' C S x S is obtained form the transition 
relation of J\f by projection. Observe that any path in Q{M) corresponds to a possible 
sequence of transitions in M since in each step, symbols under the reading heads can always 
change enabling the desired transition. Such statements are not possible for general PTM, 
since the reachability of any control-state still depends on the tape configurations and thus 
cannot be reduced to a reachability question in the induced graph. Nevertheless, for PNTM 
the following holds. 

Lemma 4.8. Given a PNTM M = (S, E, T, M, T, e, w), for any CTL* formula (ft over sets 
Fi ■ ■ ■ F n of Qx -states ■ ■ ■ Q n -states for some Qi • • • Q n C S, the set of global states s satisfying 
s\= (ft is computable. 

Proof. Observe that checking s \= (ft is equivalent to checking, in Q(M), s \= (ft' where s is 
the control-state in s and (ft' is the formula obtained from (ft by replacing all occurrences 
of Fx ■ ■ ■ F n by Qi • • • Q n respectively. It follows that the set of global states satisfying (ft is 
exactly the set of Q-states such that for any s € Q, s (= (ft' . Since G(N) is finite, the result 
follows by decidability of CTL* model-checking in finite-state systems ( |CGP99j ). □ 

A PNTM M induces a Markov chain M. = (S, P) on the set of global states. Each 
transition in A4 is also a combination of a noisy step followed by a normal step. However, 
in the noisy steps, we assume that cells not under the reading heads are not subjected to 
noise. Observe that this is different than the way noise is added in the model of |AC05] 
where, for instance, all cells are subject to noise. Intuitively, the noise doesn't affect the 
computations of the underlying Turing machine unless it changes a cell which is going to 
be visited by the reading head. Now, whether the content of that cell changes when the 
reading head reaches it or has changed in the previous steps; the resulting computation is 
the same. 

In order to compensate for the missing noise, we assume a higher noise probability for 
the cell under the head. If the cell was last visited k time units ago, then we increase the 
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PLCS 


PVASS 


PNTM 


Prob M {s lmt \= OF) = 1 


Decidable when F 
is effectively 
representable; 
see Theorem 15.61 


Decidable when F 
is defined by 
control-states ; 
see Theorem 15.31 


Decidable when F 
is defined by 
control-states; 
see Theorem 15.71 


Prob M {sinit h<>^) = 


Decidable when F 
is effectively 
representable; 
see Theorem|5.10| 


Decidable when F 
is expressible in the 
lode of |Jan90j: 
see Theorem 15.91 


Decidable when F 
is defined by 
control-states; 
by Theorem [5JJJ 



Table 3: Decidability results for qualitative reachability. 



°In Theorem 15.41 we prove that this is undecidable when F is a general upward-closed set. 

noise probability to 1 — (1 — e) k . The probability of a transition in the induced Markov 
chain is obtained by multiplying the noise probability by the probability of the normal step 
described earlier. 

Theorem 4.9. Each Markov chain induced by a PNTM Af = (S, E, r, M, T, e, w) is coarse, 
effective and finitely spanning with respect to any set of Q-states for some (JCS. 

Proof. Assume a PNTM N = (S, E, r, M, T, e, w), a set Gj C S and the induced Markov chain 
A4. Let F be the set of Q-states. Effectiveness of Ai w.r.t. F follows from the definition 
and Lemma 14.81 For any state s 6 S, if s \= 30F then there is a path in Q(Af) from the 
control-state of s to a control-state in Q. Such a path has length at most N = \G(f\f)\. Thus 
A4 has span N with respect to F. Along this path, it is possible that, in each step, each 
symbol under a reading head is subject to noise. Since in each step, M cells are subject to 
noise and each happens with probability > e, it follows that the probability of each successor 
is > (e/|r|) M . This gives the coarseness of A4. □ 

This, combined with Lemma 13.61 and Lemma 13.71 yields the following corollary. 

Corollary 4.10. Each Markov chain induced by a PNTM M = (S, E, r, M, T, e, w) is decisive 
and thus (by Lemma \3.Sfy strongly decisive with respect to any set of Q-states for some Q C S. 

5. Qualitative Reachability 

We consider the qualitative reachability problem for Markov chains, i.e., the problem if a 
given set of final states is eventually reached with probability 1, or probability 0, respectively. 

Qual_Reach 
Instance 

• A Markov chain M = (S, P) 

• A state s i n it G S 

• A set of final states F C S 

Task Decide if Prob M {s ini t \= OF) = 1 (or = 0). 

We show that, for decisive Markov chains, these qualitative questions about the Markov 
chain can be reduced to structural properties of the underlying transition graph. The 
decidability results for PLCS, PVASS and PNTM are summarized in Table [3l 
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First we consider the problem if ProbM{sinit \= OF) = 1. The following Lemma holds 
for any Markov chain and any set of states F. 

Lemma 5.1. Probj^\{si n it \= OF) = 1 implies Si n u \/= F Before F. 

Proof. If Si n it \= F Before F then there is a path tt of finite length from Si n u to some state 
in F s.t. F is not visited in tt. The set of all continuation runs of the form tttt' thus has a 
non-zero probability and never visits F . Thus ProbM( s init \= OF) < 1. □ 

The reverse implication of Lemma l5.1l holds only for Markov chains which satisfy certain 
conditions. 

Lemma 5.2. Given a Markov chain Ai and a set F such that A4 is decisive w.r.t. F , then 
we have that Si n u F Before F implies Prob^^s^ |= OF) = 1. 

Proof. From Smit \/= F Before F it follows that (s, n it (= OF) C (si n it (= OF). Therefore 
Prob_M(si n it \= <>F) = ProbM(sinit \= OF V OF). Since M. is decisive w.r.t. F, it follows 
that 1 = Prob M (s mU \= OF V OF) = Prob M {s imt (= OF). □ 

Lemma 15.21 does not hold for general Markov chains; see Remark 16.31 in Section [6l 
Now we apply these results to Markov chains derived from PVASS. Interestingly, decid- 
ability depends on whether the target set F is a set of Q-states for some Q C S or a general 
upward-closed set. 

Theorem 5.3. Given a PVASS (S,X,T, w) and a set of final states F which is the set of 
Q-states for some Q C S. Then the question Prob_M (sim* |= OF) = 1 is decidable. 

Proof. Since any set F of Q-states is upward-closed, we obtain from Corollary 14.41 that the 
Markov chain derived from our PVASS is decisive w.r.t. such F. Thus, by Lemma 15.11 
and Lemma 15.21 we obtain ProbM (sinit \= OF) < 1 Smit \= F Before F. To decide 

the question Si n u (= F Before F, we construct a modified PVASS (S,X,T',w') by removing 
all outgoing transitions from states q € Q. Formally, T' contains all transitions of the form 
(si, op, S2) € T with si ^ Q and w'(t) = w(t) for t G TDT'. Furthermore, to avoid deadlocks, 
we add to each state in Q a self-loop which does not change the values of the variables and 
whose weight is equal to one. It follows that Si n u \= F Before F in (S, X, T, w) iff Si n a \= 30F 
in (S,X,T',w'). 

So we obtain that Probj^ {si n u \= OF) = 1 in (S,X,T,w) iff F is not reachable in the 
VASS (S,X,T'). 

The condition if F is reachable in the VASS (S,X,T') can be checked as follows. Since, 
F is upward-closed, the set of predecessors Pre*(F) is upward-closed and can be effectively 
constructed by Remark 14.21 Thus the set F = Pre*(F) can be effectively described by a 
formula $ in the constraint logic of [ Jan90j . Finally, by Theorem 14. 1\ it is decidable if there 
is a reachable state in F (i.e., satisfying $). □ 

The situation changes if one considers not a set of Q-states as final states F, but rather 
some general upward-closed set F (described by its finitely many minimal elements). In 
this case one cannot effectively check the condition Si n u \= F Before F. 

Theorem 5.4. Given a PVASS V = (S,X,T,w) and an upward-closed set of final states F 
(represented by its finitely many minimal elements), then the question Probj^i (si n u \= OF) = 
p is undecidable for any p E (0, 1] . 
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We will need the following definition for the proof. 

Definition 5.5. We define a PVASS which weakly simulates a Minsky [Min67j 2-counter 
machine. Since this construction will be used in several proofs (Theorem 15.41 and Theo- 
rem [97T]), it contains a parameter x > which will be instantiated as needed. 

Consider a deterministic Minsky 2-counter machine M with a set of control-states K, 
initial control-state ko, final accepting state k acc , two counters c\ and C2 which are initially 
zero and the usual instructions of increment and test-for-zero-decrement. For technical 
reasons we require the following conditions on the behavior of M. 

• Either M terminates in control-state k acc , or 

• M does not terminate. In this case we require that in its infinite run it infinitely often 
tests a counter for zero in a configuration where the tested counter contains a non-zero 
value. 

We call a counter machine that satisfies these conditions an IT-2-counter machine (IT 
for 'infinitely testing'). Any 2-counter machine M' can be effectively transformed into an 
equivalent IT-2-counter machine M by the following operations. After every instruction of 
M' we add two new instructions: First increment c\ by 1 (thus it is now certainly nonzero). 
Then test c\ for zero (this test always yields answer 'no'), decrement it by I (so it has its 
original value again), and then continue with the next instruction of M'. So M is infinitely 
testing and accepts if and only if M' accepts. Since acceptance is undecidable for 2-counter 
machines [Min67] . it follows that acceptance is also undecidable for IT-2-counter machines. 

We construct a PVASS V = (S,X, T,w) that weakly simulates M as follows. S = K U 
{k l | k & K, i G {1, 2}}U{err} and X = {ci, c^}. For every instruction k\ : q := Cj + 1; goto ki 
we add a transition (Ari,op, k-i) to T, where op(cj) = 1 and op(cj) = for j ^ i and 
w((/ci, op, A^)) := 1- For every instruction k\ : If q = then goto k<i else c, := q — 1; goto £3 
we add the following transitions to T. 

a: (ki, op 1 , Aft) with op 1 (cj) = —1 and op 1 (c J ) = for j ^ i and w((/ci, op 1; Aft)) = f . 

0: (ki, op 2 , A; 2 ) with op 2 (cj) = for j = 1,2 and w((A;i, op 2 , k^j) = x (x > is a 
parameter of w) . 

T- (kh °Pa> ^2) with op a (cj) = for j = 1,2 and w((A$, op a , k 2 )) = 1. 

5: (k\, op b , err) with op b (cj) = — 1 and op 6 (cj) = for j 7^ i and w((k\, op b , err)) = 1. 
Finally, to avoid deadlocks in V, we add two self-loops (k acc , opi, k acc ) and (err,opi, err) 
with op^(cj) = for j = 1,2 and weight 1. 

Proof, (of Theorem I5.4D Since F is upward-closed, we obtain from Corollary 14.41 that the 
Markov chain derived from our PVASS is decisive w.r.t. F. Thus, by Lemma 15.11 and 
Lemma 15.21 we have Prob^x (smi* |= ^>F) < 1 <^=^> Si n n (= F Before F. Now we show that 
the condition Si n n (= F Before F is undecidable if F is a general upward-closed set. We use 
the IT- 2-counter machine M and the PVASS V from Def. 15.51 and instantiate the parameter 
x := 1. Let F be the set of configurations where transitions of type 5 are enabled. This set 
is upward-closed, because of the monotonicity of VASS, and effectively constructible (i.e., 
its finitely many minimal elements). It follows directly from the construction in Def. 15.51 
that a transition of type 5 is enabled if and only if the PVASS has been unfaithful in the 
simulation of the 2-counter machine, i.e., if a counter was non-zero and a 'zero' transition 
(of type pi) has wrongly been taken instead of the correct 'decrement' transition (of type 
a). 
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If the 2-counter machine M accepts then there is a run in the PVASS V which faithfully 
simulates the run of M and thus never enables transitions of type 5 and thus avoids the 
set F. Since the fc acc -states have no outgoing transitions (except for the self-loop), they are 
trivially contained in F . Thus Si n n \= F Before F. 

If the 2-counter machine M does not accept then its run is infinite. By our convention 
in Def. 15.51 M is an IT- 2-counter machine and every infinite run must contain infinitely 
many non-trivial tests for zero. Thus in our PVASS V, the set F is reachable from every 
reachable state s' which was reached in a faithful simulation of M, i.e., without visiting F 
before. Therefore in V the set F cannot be reached unless F is visited first, and so we get 
Sinit y= F Before F. 

We obtain that M accepts iff Sj„# |= F Before F iff ProbM (sinit \= OF) < 1- This 
proves the undecidability of the problem for the case of p = 1 . 

To show the undecidability for general p € (0, 1] we modify the construction as follows. 
Consider a new PVASS V which with probability p does the same as V defined above and 
with probability 1 — p immediately goes to the accepting state k acc . Then the IT-2-CM M 
accepts iff Prober (sj n # \= OF) ^ p. □ 

Notice the difference between Theorem 15 .31 and Theorem 15. 41 in the case of p = 1. Unlike 
for non-probabilistic VASS, reachability of control-states and reachability of upward-closed 
sets cannot be effectively expressed in terms of each other for PVASS. 

Theorem 5.6. Consider a PLCS C and an effectively representable set of final states F. 
Then the question Probjn (simt \= OF) = 1 is decidable. 

Proof. By Corollary 14.71 the Markov chain induced by C is decisive w.r.t. such F. Thus we 
obtain from Lemma f5.1l and Lemma f5.2l that ProbM {sinit (= OF) = 1 iff Si n u y= F Before F. 
This condition can be checked with a standard construction for LCS (from |AR03] ) as 
follows. First one can effectively compute the set F = Pre*(F) using the techniques from, 
e.g., [AJ96]. Next one computes the set X of all configurations from which it is possible to 
reach F without passing through F. This is done as follows. Let Xq := F and Xi+i := X{ | 
U (Pre(Xi)nF) j. Since all X% are upward-closed, this construction converges at some finite 
index n, by Higman's Lemma |Hig52| . We get that X = X n is effectively constructible. 
Finally we have that ProbM (sjn# |= OF) = 1 iff Si n u £ X, which can be effectively checked. 

□ 

Notice that, unlike in earlier work [AR03, BS03], it is not necessary to compute the 
finite attractor of the PLCS-induced Markov chain for Theorem 1 5. 61 It suffices to know that 
it exists. For PLCS it is very easy to construct the finite attractor, but this need not hold 
for other classes of systems with attractors. However, the criterion given by Lemma l5.1l and 
Lemma 15.21 always holds. 

Theorem 5.7. For a PNTM M = (S, E, T, M, T, e, w), the question Prob M (s init \= OF) = 1 
is decidable for any set F of Q-states for some Q C S. 

Proof. By Corollary 14.101 we obtain that the Markov chain M derived from is decisive 
w.r.t. F. This combined with Lemma 15.11 and Lemma 15.21 yields Probj^i (si n u \= OF) < 
1 <^=^ Si n it \= F Before F. Observe that since F is a set of Q-states, we obtain by 
Lemma 14.81 that we can compute a set Q' C S such that F = Q'-states. Since F and F 
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are sets of Q-states and Q'-states respectively, it follows by Lemma 14.81 that the question 
Sinit \= F Before F is decidable. This gives the result. □ 

Now we consider the question Probst {si n n \= OF) = 0. The following property trivially 
holds for all Markov chains. 

Lemma 5.8. Prob M (s mit (= OF) = iff s imt ty= 30F. 

The reachability problem for Petri nets/VASS is decidable |May84| , and the following 
result is a direct consequence of Lemma 15.81 and Theorem 14.11 

Theorem 5.9. Given a PVASSV = (S,X,T,w) and a set of final states F which is express- 
ible in the constraint logic of [Jan90] (in particular any upward-closed set, any finite set, 
and their complements), then the question Probjvi (sinit \= OF) = is decidable. 

From Lemma f5,8l and the result that for LCS the set of all predecessors of any effectively 
representable set can be effectively constructed (e.g., |AJ96j ). we get the following. 

Theorem 5.10. Given a PLCS C and a set of final states F which is effectively repre- 
sentable, then the question Probj^ (sj n ^ |= OF) = is decidable. 

By Lemma 14.8} we obtain the following. 

Theorem 5.11. Given a PNTM M = (S, E, T, M, T, e, w) and a set F of Q-states for some 
Q C S, then the question Probj^i (sj n jf \= OF) = is decidable. 



6. Qualitative Repeated Reachability 

Here we consider the qualitative repeated reachability problem for Markov chains, i.e., the 
problem if a given set of final states F is visited infinitely often with probability 1, or 
probability 0, respectively. 

Qual_Rep_Reach 
Instance 

• A Markov chain M = (S, P) 

• A state Smit € S 

• A set of final states F C S 

Task Decide if Prob M {s init \= OOF) = 1 (or = 0). 

We show that, for decisive Markov chains, these qualitative questions about the Markov 
chain can be reduced to structural properties of the underlying transition graph. The 
decidability results for PLCS, PVASS and PNTM are summarized in Table HI 

small 

First we consider the problem if Prob^. (si n it \= DOF) = 1. The following lemma holds 
for any Markov chain and any set of states F. 

Lemma 6.1. Let A4 = (S, P) be a Markov chain and F C S. Then Probj^ {si n u \= DOF) = 
1 implies Si n n \= VCGOF. 

Proof. Suppose that Si n n y= VDBOF. Then Si n a \= 30VD-F. Thus there exists a finite 
path 7r starting from Si n u leading to a state s s.t. s \= VCHF. The set of all Sj n ji-runs of 
the form irir' (for any ir') has non-zero probability and they all satisfy -iDOF. So we get 
that Prob M (sinit (= °OF) < 1. □ 
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tit no 


Jr" VASS 


JtTn 1 IV1 


Prob M (s init \= OOF) = 1 


Dccidable when 
F is effectively 
representable; 
see Theorem 16.51 


Decidable when 
F is upward- 
closed; 

see Theorem 16.41 


Decidable when 
F is defined by 
control-states; 
see Theorem 16.61 


Prob M {s init \= OOF) = 


Dccidable when 
F is effectively 
representable 
see Theorem 16.131 


Open problem 


Decidable when 
F is defined by 
control-states; 
see Theorem 16.141 



Table 4: Decidability results for qualitative problems of repeated reachability. 



The reverse implication of Lemma [6. II does not hold generally, but it is true for strongly 
decisive Markov chains. 

Lemma 6.2. Given a Markov chain M and a set F such that A4 is strongly decisive w.r.t. 
F , then we have that Si n n \= VD30F implies Prob M (s imt \= OOF) = 1. 

Proof. We show that Prob_M{si n it \= OCHF) = which implies the result. 

If Sinit \= VnBOF then every state s reached by runs in (sj„^ (= Od-F) satisfies 
s \= 30F. Therefore (s mU \= OO^F) C (s mU \= □-.£). 

Since the Markov chain is strongly decisive, Prob_M{ s init \= OF V OOF) = 1 and 
thus Probj^{si n it \= O—iF A OCHF) = 0. It follows from the inclusion above that = 
Prob M (si n it h A OCHF) = Prob M (s imt (= n-iF) > Prob M {s imt \= OU^F). □ 

Remark 6.3. Neither Lemma 15.21 nor Lemma 16.21 hold for general Markov chains. A 
counterexample is the Markov chain Ai = (S, P) of the 'gambler's ruin' problem |Fel66] 
where S = N, P(i, i + 1) := x, P(i, i - 1) := 1 - a; for i > 1 and P(0, 0) = 1 and F := {0}, 
for some parameter x G [0, 1]. It follows that F = for x < 1. If x G [0,1/2] then 
Prob M (1 |= OF) = 1 and for x > 1/2 one has Pto&m (1 \= OF) = (1 - 

For a; G (1/2, 1) one has that 1 ^ F Before F, but Prob M (1 |= OF) = (1 - x)/x < 1. 
Similarly, although 1 |= VD30F, one still has Prob M (1 |= OOF) < Prob M (1 \= OF) = 
(l-x)/x<l. 

Now we show that it is decidable if a PVASS almost certainly reaches an upward-closed 
set of final states infinitely often. 

Theorem 6.4. Let V = (S,X,T,w) be a PVASS and F an upward-closed set of final states. 
Then the question Probjn (si n it \= OOF) = 1 is decidable. 

Proof. Since F is upward-closed, we obtain from Corollary 14.41 that the Markov chain de- 
rived from our PVASS is strongly decisive w.r.t. F. Thus it follows from Lemma 16.11 and 
Lemma 16.21 that Probj^ {si n u (= OOF) = 1 <^=^> Sj n y (= VnBOF. This condition can be 
checked as follows. Since F is upward-closed and represented by its finitely many minimal 
elements, the set Pre*{F) is upward-closed and effectively constructible by Remark 14.21 
Then F = Pre*{F) is downward-closed and effectively representable by a formula $ in the 
constraint logic of [Jan90j. We get that Si n u \= VD30F iff Si n u \/= BOF, i.e., if there is no 
reachable state that satisfies This is decidable by Theorem 14.11 □ 
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Notice the surprising contrast of the decidability of repeated reachability of Theorem [63] 
to the undecidability of simple reachability in Theorem 15.41 
Now we show the decidability result for PLCS. 

Theorem 6.5. Consider a PLCS C and an effectively representable set of final states F. 
Then the question ProbM (si n it \= OOF) = 1 is decidable. 

Proof. By Corollary 14.71 £ induces a strongly decisive Markov chain. Thus, we obtain from 
Lemma 16.11 and Lemma 16.21 that ProbM (sma (= OOF) = 1 Si n a \= VOBOF. This 

condition can be checked as follows. First one can effectively compute the set F = Pre*(F). 
Next one computes the set X of all configurations from which it is possible to reach F, 
i.e., X := Pre*{F). Finally we have that Prob M (s inU \= OF) = 1 iff s imt (= iff 
Sinit £ X, which can be effectively checked. □ 

Similarly as in Theorem 15.61 it is not necessary to compute the finite attractor of the 
PLCS-induced Markov chain for Theorem 16.51 

Next we prove the decidability result for PNTM. 

Theorem 6.6. Let M = (S, E, T, M, T, e, w) be a PNTM and F a set of Q-states for some 
G) C S. Then the question ProbM ( s init \= OOF) = 1 is decidable. 

Proof. Since F is a set of Q-states, we obtain from Corollary 14.101 that the Markov chain 
derived from M is strongly decisive w.r.t. F. Thus it follows from Lemma [6.1l and Lemma f6.2l 
that ProbM (sinit \= OOF) = 1 -4=^> Si n u \= VcGOF. We can check this condition by 
Lemma 14.81 □ 

Now we consider the question ProbM{sinit \= OOF) = 0. We start by establishing some 
connections between the probabilities of reaching certain sets at least once or infinitely often. 
From the definitions we get the following. 

Lemma 6.7. For any Markov chain and any set of states F, we have ProbM(sinit \= 
□ OF) / implies Prob M (s init \= OF) ^ 1 

The following lemma implies that the reverse implication holds for strongly decisive 
Markov chains. 

Lemma 6.8. Given a Markov chain M which is strongly decisive w.r.t. a given set F, 
then we have that ProbM(sinU \= OOF) = 1 — ProbM^init |= OF). 

Proof. 1 - Prob M (sinit \= OF) = Prob M (s imt \= U-tF) = Prob M (sinit (= H-,F A OOF) + 
ProbM(si n u \= D—\F A On-ii 7 ). Since the Markov chain is strongly decisive, it follows that 
Prob M {sinit H Q^AODnF) = 0. Moreover, we haveJs M |= OOF) C (s inM \= O-iF). 
Therefore, 1 - Prob M {si n it \= OF) = Prob M {s in it \= D->F) = Prob M {sinit \= OOF). □ 

There is also a correspondence of the condition ProbM( s init \= OOF) / to a property 
of the underlying transition graph. 

Lemma 6.9. For any Markov chain Ai and any set of states F, Si n a \= 30F implies 
Prob M (sinU \=OF)^l. 

Proof. If 

s init H ^OF then there exists a finite path starting from Si n n and leading to some 
state in F. Therefore we obtain ProbM( s init \= OF) > 0. Observe that any state s reached 
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by runs in (sj n jt \= OF) before reaching F satisfies s \= 30F. So we have s \= BOVCGOi* 1 
and therefore s \= 30F and thus s \= ~^F. Furthermore, every state s reached by runs in 

{sinit \= OF) after reaching F also satisfies s \= -*F, because, by definition, F cannot be 
reached from F. _ _ 

This yields {si n u \= OF) C (sj„^ |= O^F) which implies that < Prob^(si n it \= OF) < 
ProbM(sinit \= a ^F), where the first inequality follows from the argument mentioned above. 
Finally, we obtain ProbM(sinit (= OF) = 1 - ProbM(simt (= O^F) < 1 - ProbM(sinit \= 
OF) < 1. □ 

If Ai is decisive w.r.t. F then the reverse implication holds. 
Lemma 6.10. Given a Markov chain Ai and a set of states F s.t. Ai is decisive w.r.t. F, 
then the condition Prob_M{si n u \= OF) ^ 1 implies Sj n y (= 30F. 

Proof. As M is decisive w.r.t. F, it follows that ProbM( s init \= OF V OF) = 1. Since by 
assumption Prob_M(si n it \= OF) ^ 1, it follows that ProbM(sinit \= OF) > 0, which implies 
s imt \= 30F. □ 
Theorem 6.11. For any Markov chain Ai and any set F s.t. Ad is decisive w.r.t. F 

s init \= 30F => Prob M (s inU \= OF) / 1 ^ Prob M (s imt \= OOF) ± 
For any Markov chain Ai and any set F s.t. Ai is decisive w.r.t. F and w.r.t. F 

s inU h ^ Prob M {s imt \= OF) / 1 ^ Prob M (s imt (= UOF) + 

Proof. Directly from Lemma 16.71 16.81 16.91 and 16.101 □ 

Remark 6.12. Observe that decisiveness w.r.t. a given set F does not imply decisiveness 
w.r.t. F. Therefore the reverse implication of Lemma 16.91 does not hold in general. In par- 
ticular, it holds for Markov chains with a finite attractor (since they are decisive w.r.t. every 
set), but not generally when we have global coarseness. This is because global coarseness 
depends on the set of final states. Global coarseness of a Markov chain w.r.t. a certain set 
F does not imply global coarseness w.r.t. the set F. 

Now we show the decidability results for PLCS and PNTM. 

Theorem 6.13. Consider a PLCS C and an effectively representable set of final states F. 
Then the question Probjn (si n it \= OOF) = is decidable. 

Proof. By Corollary 14.71 C induces a strongly decisive Markov chain w.r.t. every set of 
states. In particular it is decisive w.r.t. F and F. Therefore, by Theorem 16.111 it suffices 

to check if Si n u 30F. Since the upward-closure of F is effectively constructible, one can 

effectively construct a symbolic representation of the set of all states which satisfy 30F 
(using the techniques from, e.g., |AJ96| ) and check if Si n a is not in this set. □ 
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Theorem 6.14. Given a PNTM M = (S, E, T, M, T, e, w) and a set F of Q- states for some 
Q C S, the question ProbM {si n u \= OOF) = is decidable. 

Proof. Observe that since F is a set of Q-states, by Lemma 14,81 we can construct the set 
Q' C S such that F is exactly the set of Q'-states. By Corollary 14.101 the Markov chain 
induced by J\f is decisive w.r.t. any set of Q"-states for some Q" C S. In particular, it is 

decisive w.r.t. Q'-states = F. By Theorem 16. 1 H it suffices to check if Sj n # 30F which is 
again decidable by Lemma 14.81 □ 

Remark 6.15. For PVASS, decidability of Probjvi {sinit \= OOF) = and the equiva- 
lent question Prob_M(sinit (= OF) = 1 is open. For an upward-closed set F the set F is 
downward-closed, but in general not a set of Q-states, and thus Theorem 15.31 does not al- 
ways apply. The question Probjvi(si n it (= OF) = 1 can certainly not be reduced to purely 
structural questions about the underlying transition system (unlike for PLCS), because it 
depends on the exact values of the probabilities, i.e., on the transition weights. 

Consider a PVASS model of the gambler's ruin problem (see Remark I6.3[) . but let now 
F := {1,2,...} (upward-closed) and thus F = {0}. We have Prob M (l \= OF) = (l-x)/x < 
1 for x > 1/2 and Probjn(l \= OF) = 1 otherwise. 

In particular, for x > 1/2, this system is decisive w.r.t. F, but not decisive w.r.t. F, 

because F = and Prob M {l (= OF V OF) = Prob M {l \= OF) < 1. 

Furthermore, for PVASS, the probability Probj^i {sinit \= OOF) cannot be effectively 
expressed in the first-order theory of the reals (H, +,*,<), as shown in Section (H Re- 
mark 

7. Approximate Quantitative Reachability 

In this section we consider the approximate quantitative reachability problem. 

Approx_Quant_Reach 
Instance 

• A Markov chain M = (S, P) 

• A state Simt £ S 

• A set of states F C S 

• A rational e > 

Task Compute a rational 9 such that 6 < Probj^ (si n u \= OF) < + e. 

We show that this problem is effectively solvable for PLCS, PVASS and PNTM, pro- 
vided that the induced Markov chain is decisive w.r.t. F. 

First, we present a path enumeration algorithm, based on [IN97] . for solving the prob- 
lem, and then we show that the algorithm is guaranteed to terminate for all instances where 
A4 is decisive w.r.t. F0. 

Given an effective Markov chain M. = (S,P), a state Si n u G S, a set F and a positive 
e G Q>o, the algorithm constructs (a prefix of) the reachability-tree, from Sj n #, in a breadth- 
first fashion. The nodes of the tree are labeled with pairs (s, r) where s G S and r is the 



1 Termination of the algorithm is stated (without giving a proof) in |Rab03| for the case of Markov chains 
which contain a finite attractor. 
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probability of traversing the path from the root to the current node. Every node in the tree 
is labeled with a probability. This probability is the product of the probabilities of all the 
transitions in the path from the root to the node. The algorithm maintains two variables 
Yes and No which accumulate the probabilities by which the set F is certainly reached (and 
certainly not reached, respectively). Each step of the algorithm can be implemented due 
to the effectiveness of M. The algorithm runs until we reach a point where the sum of Yes 
and No exceeds 1 — e. 



Algorithm 1 - Approx_Quant_Reach 




Input 




A Markov chain M. = (S,P), a state Sj n j 4 G S 


a set F C S and a positive e G Q>o 


such that M. is effective w.r.t. F. 




Return value 




A rational 9 such that 6 < Probjvi {sinit \= OF) 


< # + e 


Variables 




Yes, No: Q (initially all are set to 0) 


store: queue with elements in S x Q 




begin 




1. store := (s init , 1) 




2. repeat 




3. remove (s,r) from store 




4. if s E F then Yes := Yes + r 




5. else if s G F then No := No + r 




6. else for each s' € Post(s) 




7. add (s', r ■ P(s, s')) to the end of store 




8. until Yes + No > 1 - e 




9. return Yes 




end 





We require that the Markov chain is effective w.r.t. F so that the condition s G -F in line 5. 

can be effectively checked. 

Let Yesi(M ! Sinu) denote the value of variable Yes after the algorithm has explored 

the reachability-tree with root Si n a up to depth j (i.e., any element (s,r) in store is such 
<j-\-i 

that Sj„jf - — ► s). We define No J (A4, Si n u) in a similar manner. First we show partial 
correctness of Algorithm [H 

Lemma 7.1. If Algorithm^ terminates at depth j then 

Yes j {M, Sinn) < Prob M {s im t \= OF) < Yes j (M, s in it) + e 
Proof. It is straightforward to check that for each j > we have 

Yes 3 {M, Sinn) < Prob M (sinit \= OF) 

and 

No j (M, s^it) < Prob M (s imt \= F Before F) 

We notice that 

Prob M (sinit h ^-F) < 1 - Prob M (s init \= F Before F) 
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It follows that 

Yes j {M,s imt ) < Prob M (s init \= OF) < 1 - No j (M, s init ) 

The result follows from the fact that Yes J (M, s init ) + No 3 (M i s init) ^ 1 ~~ e when the 
algorithm terminates. □ 

Lemma 7.2. Algorithm^ terminates in case the Markov chain A4 is decisive w.r.t. F. 

Proof. Since Ai is decisive we have Probjvi{si n it \= OFVOF) = 1. Therefore lim^oo ( Yes 1 + 
No 3 ) = 1, which implies termination of the algorithm. □ 

From Lemma 17.11 and Lemma 17.21 it follows that Approx_Quant_Reach is solvable 
for Markov chains which are globally coarse w.r.t. the target set and for Markov chains 
which contain a finite attractor. This, together with Theorem 14.31 and Theorem 14.61 yield 
the following theorems. 

Theorem 7.3. Approx_Quant_Reach is solvable for PVASS in case F is upward-closed. 

Theorem 7.4. Approx_Quant_Reach is solvable for PLCS in case F is effectively rep- 
resentable. 

Theorem 7.5. Approx_Quant_Reach is solvable for PNTM in case F is a set of te- 
states. 



8. Approximate Quantitative Repeated Reachability 

In this section we approximate the probability of reaching a given set of states infinitely 
often, i.e., we compute arbitrarily close approximations of Probjn (si n it \= OOF). 

Approx_Quant_Rep_Reach 
Instance 

• A Markov chain M = (S, P) 

• A state Simt £ S 

• A set F C S 

• A rational e > 

Task Compute a rational 9 such that 6 < Probj^i (sinit \= OOF) < 9 + e. 

We present an algorithm which is a modification of Algorithm Q] (in Section [7|) and show 
that it is guaranteed to terminate for all Markov chains which are decisive w.r.t. both F 
and F. 
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Input 




A Markov chain M. = (S,P), a state Sinit € S, 


a set F C 5 and a positive e G Q>o 


such that A4 is effective w.r.t. F and F . 




Return value 




A rational 9 such that < ProbM (si n it \= OOF) 


< 9 + e 


Variables 




Yes, No: Q (initially all are 


set to 0) 


store: queue with elements in S x Q 




KpCFl 11 
vJ 111 




1. blOl t .— ybmiti 1 ) 




2 rpnpaf 




3. remove (s,r) from store 




4. if s G F then Yes := Fes + r 




5. else if s G F then No := A/o + r 




6. else for each s' 6 Post(s) 




7. add (s', r • P(s, s')) to the end of store 




8. until Yes + iVo > 1 - e 




9. return Yes 




end 





We require that the Markov chain is effective w.r.t. F and F so that the conditions 

s € -F (in line 5.) and s £ -F (in line 4.) can be effectively checked. 

We define Yes 3 (M., Sinit) and No 3 (A4, Sinit) as the values of the variables Yes and No 
after the algorithm has explored the reachability-tree with root Si n it up-to depth j, similarly 
as for Algorithm [TJ The following Lemma shows the partial correctness of Algorithm [2j 



Lemma 8.1. For a Markov chain A4 and a set F such that M is strongly decisive w.r.t. 
F, if Algorithm^ terminates at depth j then 

Yes 3 (M, Sinu) < Prob M (s init (= OOF) < Yes 3 (M, s init ) + e 

Proof. If Algorithm [2] reaches some state s € F (at line 4.) then we have s \= \/030F. 
Since A4 is strongly decisive, it follows from Lemma [6 . 2 1 that ProbM (s \= OOF) = 1. Thus, 
for each j > 0, we have Yes 3 (A4, Sinit) < ProbM (sinit \= OOF). Similarly, if the algorithm 
reaches some state s € F (at line 5.) then we have (s |= OOF) = 0. Thus, for each j > 0, 
we have No 3 (M,s in it) < Prob M (s ini t ty= OOF) = 1 - Prob M (s init \= OOF). It follows that 
Yes 3 (M,s im t) < Prob M {sinit \= OOF) < 1 - No 3 (M 

i s init)- The result follows from the 
fact that Yes 3 (M, s inU ) + No 3 (M, Sinit) ^ 1 — £ when the algorithm terminates. EH 

Lemma 8.2. Algorithm^ terminates if M is decisive w.r.t. F. 

Proof. Since M. is decisive w.r.t. F, we have ProbM (^ s init \= ^F V OF^j = 1. It follows 

that linij^oo ( Yes 3 + No 3 ) = 1, which implies termination. □ 

Note that Algorithm [2] only works for Markov chains which are decisive w.r.t. both F 
and F. Decisiveness w.r.t. F is required for termination (Lemma 18. 2|) . while decisiveness 
w.r.t. F is required for correctness (Lemma 18. ip . 
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Now we show the computability results for PLCS and PNTM. 

Theorem 8.3. Approx_Quant_Rep_Reach is solvable for PLCS in case F is effectively 
representable. 

Proof. By Theorem 14.61 a Markov chain induced by a PLCS is decisive w.r.t. every set, 
in particular w.r.t. F and F. Thus it follows from Lemma 18.11 and Lemma 18.21 that 
Algorithm [2] solves Approx_Quant_Rep_Reach for PLCS. □ 

Theorem 8.4. Approx_Quant_Rep_Reach is solvable for PNTM in case F is a set of 
Q-states. 

Proof. Assume a PNTM M = (S, E, r, M, T, e, w) and a set F of Q-states for some Q C S. 
Since F is the set of Q-states, it follows by Lemma [4.81 that F is also a set of Q'-states where 
Q' C S. By Theorem 14.91 the Markov chain induced by a PNTM is decisive w.r.t. every set 
of Q"-states, in particular w.r.t. F and F. Thus it follows from Lemma [8. II and Lemma [8.21 
that Algorithm [2] solves Approx_Quant_Rep_Reach for PNTM. □ 

A similar result for PVASS would require the explicit assumption that the induced 
Markov chain is decisive w.r.t. F and F. It is not sufficient that F is upward-closed, because 
this only implies decisiveness w.r.t. F, not necessarily w.r.t. F. (See the counterexample in 
Remark 16.151 ) 

9. Exact Quantitative Analysis 



In this section we consider the Exact Quantitative Reachability Analysis Problem, defined 
as follows. 



Exact_Quant_Reach 




Instance 




• A Markov chain A4 = 


(S,P) 


• A state Sinn £ S 




• A set of states F C S 




• A rational v 




Task Check whether Probj^ 


{s im t h= of) > 



By Theorem 15.41 Exact_Quant_Reach is undecidable for PVASS and upward-closed 
sets F. If F is a set of Q-states then decidability of Exact_Quant_Reach is open for 
PVASS, PLCS and PNTM. 

However, for PVASS, PLCS and PNTM, we show that the probability Probj^ {smit \= OF) 
(and thus the question of Exact_Quant_Reach ) cannot be effectively expressed in a uni- 
form way in the first-order theory of the reals (H, +,*,<), or any other decidable logical 
theory with first-order quantification. By 'expressed in a uniform way' we mean that pa- 
rameters of the system (e.g., transition weights for PVASS, the message loss probability for 
PLCS or the noise parameter for PNTM) should be reflected directly by constants in the 
constructed formula (see the remarks at the end of this section for details). This negative 
result for PVASS, PLCS and PNTM is in contrast to the situation for probabilistic push- 
down automata (probabilistic recursive systems) for which this probability can be effectively 
expressed in (R, +, *, <) [EKM041 IEKM061 IEY05bl IEE04] . 
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Theorem 9.1. Let V = (S,X, T, w) be a PVASS and wi,... , w n £ Q the constants used in 
the transition weight function w. Let F be the set of Q-states for some Q C S. 

It is impossible to effectively construct a (R, +,*,<) formula $(p, wi, . . . , w n ) with pa- 
rameters Wj andp which expresses the probability Probj^\ (sinit (= OP)> *-e., wi, . . . ,w n ) = 
irae ijff p = Profe^ (s init \= OF). 

Proof. We assume the contrary and derive a contradiction. Consider the IT-2-CM M and 
the PVASS V with parameter x from Def. 15.51 i- e -> let wi = a; and w, = 1 for i > 1. Let P be 
the set of err-states. Suppose that one could effectively construct the (Hi, +, *, <)-formula 
<3?(p, wi, . . . ,w n ) = $>(p,x) with the required properties. 

If a counter is tested for zero then, in our weak simulation by the PVASS, there are 
two cases: 

• If the counter contains zero, then only one transition (the one of type j3) is enabled and the 
simulation is faithful. After firing transition /?, only transition 7 (but not 5) is enabled. 

• If the counter does not contain zero then two transitions, a and /3, are enabled. The 
probability of choosing a, the faithful simulation, is 1/(1 + x) and the probability of 
choosing (3 (the wrong transition in this case) is x/(l + x). If f3 is fired then both 7 and 
S are enabled. 

If the IT-2-counter machine M accepts after a finite number L of steps, then it can 
make at most L tests for zero. Thus the probability of ever choosing the wrong transition is 
bounded from above by 1 — 1/(1 + x) L . A transition of type S, leading to the err-state can 
only be taken if a wrong transition has been taken first. Thus the probability of reaching 
the err-state is also bounded from above by 1 — 1/(1 + x) L . For x — > this probability 
converges to 0. Thus we have 3x > 3p (3>(p, x) A p < 1/10). 

If the IT-2-counter machine M does not accept then its infinite run will contain infinitely 
many tests for zero on counters which are non-zero by Def. 15.51 In each of those tests, the 
chance of firing the wrong transition (type /3) is x/(l + x) > 0. Thus it will happen 
eventually with probability 1. If the wrong transition has been fired then the probability 
of going to the err-state by the next transition is 1/2 (competing enabled transitions 7 
and 5 of weight 1 each). Thus the probability of reaching the err-state is at least 1/2, i.e., 
Probjv[ (sinit \= OF) > 1/2, regardless of the value of x (provided x > 0). 

It follows that M accepts if and only if 

3x > 03p(®(p,x) Ap < 1/10) 

If the formula $(p, x) was effectively constructible, then the formula 3x > Bp (<3?(p, x) Ap < 
1/10) would also be effectively constructible and the question would be decidable, because 
of the decidability of (H, + ,*,<) [Ta r51j . This is a contradiction, since acceptance of M is 
undecidable. □ 

Remark 9.2. In Theorem 19.11 the set P is the set of err-states, i.e., an upward- and 
downward-closed set. However, the result holds just as well if P is a single configuration. 
To show this, it suffices to modify the construction as follows. Add two new transitions 
(err, op^, err) with weight 1 and opj(cj) = —1 and op i (c J -) = for i 7^ j and i, j £ {1, 2}. The 
only other possible transition in err-states is the self-loop which changes nothing. So almost 
every run starting in an err-state will eventually reach (err, 0,0). Thus the probability of 
eventually reaching configuration (err, 0, 0) is the same as that of eventually reaching some 
err-state. 
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For PLCS, the result corresponding to Theorem 19, II would be trivial if one allowed the 
case that the message loss probability A is zero, because the reachability problem for de- 
terministic non-lossy non-probabilistic FIFO-channel systems is undecidablc [BZ83j (unlike 
for VASS |May84|). The following theorem shows a stronger non-expressibility result even 
for the restricted case of A > 0. 

Theorem 9.3. Let £ = (S, C, M, T, A, w) be a PLCS with message loss probability A > 
and F a set of Q-states for some Q C S. Then it is impossible to effectively construct 
a (H, +, *, <) formula A) with parameters A > and p which expresses the prob- 
ability Probj^i (si n it |= OF), i.e., such that for any A > one has &(p, A) = true iff 
p = Prob M (Sinn \= OF). 

Proof. We assume the contrary and derive a contradiction. It is known that the termination 
problem for deterministic non-lossy non-probabilistic FIFO-channel systems £' is undecid- 
able [BZ83| . Given such a FIFO-channel system £' (and the contrary of our theorem) 
we will construct a PLCS £ and a (R, +, *, <) formula ^ s.t. ^ is true if and only if £' 
terminates. 

Consider a deterministic non-lossy non-probabilistic FIFO-channel system £' which 
starts in control-state Si n n and the empty channel. Let k acc be the final accepting control- 
state of £' . £' either eventually reaches the final accepting control-state k acc and terminates, 
or continues forever. We construct the PLCS £ by modifying £' as follows. First we add a 
new control-state err to the system. Then, for every transition t\ of £' we add an additional 
transition t<i to £. Transition t<i is almost identical to t±, except that its target control-state 
is err. We assign the same transition weight 1 to every transition in £. Thus, in every 
single step, the system £ has probability 1/2 of going to an err-state. In order to avoid 
deadlocks, we add self-loops to the control-states k acc and err. These are the only possible 
transitions from k acc and err. In particular, it is impossible to get to an err-state from any 
or vice- versa. Finally, we add the message loss probability A > to £, i.e., in 
every step any message in transit is lost with probability A. Let F be the set of fe acc -states. 

If £' terminates, then it terminates in a finite number L of steps. Since one starts with 
the empty channel, the maximal number of messages in transit at any time in this run is 
bounded from above by L. The PLCS £ can imitate this run of £' (however, £ also has many 
other possible runs). The probability that none of the (at most L) messages is lost in any 

single step is bounded from below by (1 — A) L . Thus Probj^ {si n u \= OF) > ((1 — X) L ■ |) . 
This is the probability that £ faithfully simulates the behavior of £' . No messages are lost 
and all transitions leading to err-states are avoided. Now let e := (0.9) L (0.5) L . It follows 
that 

3e > 0. VA : < A < 0.1 Prob M (s init \= OF) > e 
Now we consider the case that £' does not terminate. Let G be the set of all err-states. In 
particular, G C F. Since G is not reachable from F either, we have Probj^ (si n it \= —*OF) > 
Prob_M (si n it \= OG) . Consider all those runs of £ which do not lose any messages in the 
first steps and reach G after at most steps. These runs faithfully simulate the first N 
steps of £', unless they go to an err-state. In particular, they do not reach k acc , because £' 
does not reach k acc , since £' does not terminate. We have 

Prob M (s m u h= OG) > (1 - A)^ (1 - (0.5)") 
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Now let N := (j) f . We obtain 

lim Prob M {s inU \= OG) = 1 

A — >0 

and thus 

lim Prob M (sinit \= OF) = 

A — >0 

and finally 

-i (3e > 0. VA : < A < 0.1 Prob M {s init (= OF) > e) 

Therefore, £ terminates if and only if 

3e > 0. VA : < A < 0.1 Prob M (s init (= OF) > e 

Assume that for C the (R, +, *, <)-formula &(p,\) was effectively constructible. Then C! 
terminates if and only if 

^ := 3e > 0. VA : < A < 0.1 3p ($(p, A) A p > e) 

Since (R, +, *, <) is decidable [Tar51j one can decide if ^ is true and thus if £ terminates. 
Contradiction. □ 

The proof of the corresponding result for PNTM is similar to the one for PLCS. Note 
that the following Theorem 19.41 also holds for NTM, since the proof does not use the exten- 
sions of PNTM over NTM. 

Theorem 9.4. Let N = (S, E, T, M, T, e, w) be a (P)NTM with noise probability e > 
and F a set of Q-states for some Q C S. Then it is impossible to effectively construct 
a (R, +,*,<) formula Q(p,e) with parameters e > and p which expresses the proba- 
bility Probj^ ( s init \= OF), i.e., such that for any e > one has &(p,e) = true iff p = 
Prob M (Sinn \= OF) . 

Proof. We assume the contrary and derive a contradiction. In the proof, we start form a 
Turing machine A/ 7 . Then we construct an PNTM M and a (R, +, *, <)-formula s.t. is 
true if and only if M' halts. The contradiction follows from the undecidability of the halting 
problem for Turing- machines [HU79]. 

Assume a single-tape Deterministic Turing machine A/ 7 that starts in an initial control- 
state so with an input word loq. Once started, the machine either reaches a control-state 
and halts, or runs forever. We derive a PNTM M from M' as follows. First, we add a 
control-state s e to the system and a special symbol a to the tape alphabet. Then, from 
every state, we add a transition that reads a and moves to s e where it halts. We add loops 
to s e and Sh to avoid deadlocks. Finally, we assume that M has a noise probability e > 
and all the transitions have weight 1. 

Assume that the runs of M start all from an initial (global) state Si n a where the control- 
state is so and the initial word is loq. Let F be the set of s^-states. If AT' halts then it 
halts after a finite number of steps L. The runs which faithfully imitate those of A/"' avoid 
noise in each step. Assume that the computation reached a step where the cell under the 
head was visited k time units ago. Then, the probability to avoid the noise and move to the 
next configuration is exactly 1 — ^2 a > er 1 fc^ = (1 — e) k . Observe that this probability is 

bounded from below by (1 — e) L . Therefore, it follows that Probj^ (si n it \= OF) > (1 — e) L2 . 
Thus, by taking 9 = 0.9 L we obtain 

39 > 0. Ve : < e < 0.1 Prob M (s init (= OF) > 9. 
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Suppose that J\f' does not halt and continues running forever. Let G be the set of 
s e -states. Observe that F is not reachable from G. Similarly, by reasoning about runs of M 
which faithfully imitate those of TV' the first ./V steps, we obtain that ProbM (si n u \= ^OF) > 
Prob M [s imt (= OG) > (l-e)^ 2 . For N = (i) 1/4 , it follows that lim e ^ Prob M {s init \= OG) = 
1. As a consequence, we obtain lim^o ProbM (smit \= OF) = 0. Therefore, we have 

(36 > 0. Ve : < e < 0.1 Prob M (s imt \= OF) > 9) . 

Finally we obtain that M' halts if and only if 

36 > 0. Ve : < e < 0.1 Prob M (s init \= OF) > 6. 

Suppose that for J\f, there exists a (1R, + , *, <)-formula <&(p, e). Then M' terminates if and 
only if 

* := 39 > O.Ve : < e < 0.13p($(p,e) Ap>6) 
which is a contradiction. □ 

Remark 9.5. In the constructions for Theorem 19. 1\ Theorem 19.31 and Theorem 19.41 only 
states in F can be reached from F. Hence ProbM {sinit \= OF) = Probj^ (sj n ii \= ^OF) 
in these constructions. Thus, neither for PVASS, nor for PLCS or PNTM the probability 
ProbM {sinit \= OOF) can be expressed effectively in (M, +, *, <). 

To summarize, the results of this section only show that for PVASS, PLCS and (P)NTM 
the probabilities ProbM {sinit (= OF) and ProbM {sinit \= OOF) cannot be effectively ex- 
pressed in (H, +,*,<) in the same uniform way as for probabilistic pushdown automata 
(see [EKM041 IEKM061 IEY05bl [EE04 jV 

The following three scenarios are not ruled out for PVASS, PLCS, or PNTM. It is an 
open question which of them is true for each class of models. 

(1) The probability ProbM (sinii (= OF) is effectively expressible in (K, +, *, <), but not in 
a uniform way. For example, the parameters of the (PVASS, PLCS or PNTM) system 
would influence the formula in a complex (but still computable) way such that it is not 
possible to quantify over them in the logic. 

(2) The probability ProbM (sinit (= OF) is expressible in (H, +,*,<), but the formula is 
not effectively constructible. 

(3) The probability ProbM ( s init \= OF) is not generally expressible in (H, +,*,<), i.e., 
there is no corresponding formula at all. 



10. Conclusions and Future Work 

We have defined the decisiveness property for infinite-state Markov chains. Some other 
abstract conditions like the existence of a finite attractor or global coarseness imply the 
decisiveness property. In particular, several classes of infinite Markov chains which are 
derived from program-like probabilistic system models like PVASS, PLCS and PNTM satisfy 
these properties. 

We have studied the qualitative and quantitative (repeated) reachability problem for 
decisive Markov chains. Our main results are that qualitative questions (i.e., if some prob- 
ability is or 1) can often be reduced to questions about the underlying infinite transition 
graph of the systems, while for quantitative questions a simple path enumeration algorithm 
can be used to approximate the probabilities arbitrarily closely. 
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A surprising result was that reachability of control-states and reachability of upward- 
closed sets cannot be effectively expressed in terms of each other for PVASS, unlike for 
normal VASS (Section [5]). Furthermore, for probabilistic systems, reachability is not always 
easier to decide than repeated reachability (Theorems 15.41 and I6.4[) . 

Open questions for future work are the decidability of qualitative reachability problems 
for Markov chains with downward-closed sets of final states, and an algorithm to approxi- 
mate quantitative repeated reachability in PVASS. Furthermore, the decidability of exact 
quantitative questions like Prob_M (si n u (= OF) > 0.5 is still open for PVASS, PLCS and 
(P)NTM. 
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